MOCNN: A Multiscale Deep Convolutional Neural Network for ERP-Based Brain-Computer Interfaces.

Event-related potentials (ERPs) reflect neurophysiological changes of the brain in response to external events and their associated underlying complex spatiotemporal feature information is governed by ongoing oscillatory activity within the brain. Deep learning methods have been increasingly adopted for ERP-based brain-computer interfaces (BCIs) due to their excellent feature representation abilities, which allow for deep analysis of oscillatory activity within the brain. Features with higher spatiotemporal frequencies usually represent detailed and localized information, while features with lower spatiotemporal frequencies usually represent global structures. Mining EEG features from multiple spatiotemporal frequencies is conducive to obtaining more discriminative information. A multiscale feature fusion octave convolution neural network (MOCNN) is proposed in this article. MOCNN divides the ERP signals into high-, medium-and low-frequency components corresponding to different resolutions and processes them in different branches. By adding mid-and low-frequency components, the feature information used by MOCNN can be enriched, and the required amount of calculations can be reduced. After successive feature mapping using temporal and spatial convolutions, MOCNN realizes interactive learning among different components through the exchange of feature information among branches. Classification is accomplished by feeding the fused deep spatiotemporal features from various components into a fully connected layer. The results, obtained on two public datasets and a self-collected ERP dataset, show that MOCNN can achieve state-of-the-art ERP classification performance. In this study, the generalized concept of octave convolution is introduced into the field of ERP-BCI research, which allows effective spatiotemporal features to be extracted from multiscale networks through branch width optimization and information interaction at various scales.

An Evaluation of Hybrid Deep Learning Models for Classifying Multiple Lower Limb Actions.

Brain-computer Interfaces (BCIs) interpret electroencephalography (EEG) signals and translate them into control commands for operating external devices. The motor imagery (MI) paradigm is popular in this context. Recent research has demonstrated that deep learning models, such as convolutional neural network (CNN) and long short-term memory (LSTM), are successful in a wide range of classification applications. This is because CNN has the property of spatial invariance, and LSTM can capture temporal associations among features. A combination of CNN and LSTM could enhance the classification performance of EEG signals due to the complementation of their strengths. Such a combination has been applied to MI classification based on EEG. However, most studies focused on either the upper limbs or treated both lower limbs as a single class, with only limited research performed on separate lower limbs. We, therefore, explored hybrid models (different combinations of CNN and LSTM) and evaluated them in the case of individual lower limbs. In addition, we classified multiple actions: MI, real movements and movement observations using four typical hybrid models and aimed to identify which model was the most suitable. The comparison results demonstrated that no model was significantly better than the others in terms of classification accuracy, but all of them were better than the chance level. Our study informs the possibility of the use of multiple actions in BCI systems and provides useful information for further research into the classification of separate lower limb actions.

Novel channel selection model based on graph convolutional network for motor imagery.

Multi-channel electroencephalography (EEG) is used to capture features associated with motor imagery (MI) based brain-computer interface (BCI) with a wide spatial coverage across the scalp. However, redundant EEG channels are not conducive to improving BCI performance. Therefore, removing irrelevant channels can help improve the classification performance of BCI systems. We present a new method for identifying relevant EEG channels. Our method is based on the assumption that useful channels share related information and that this can be measured by inter-channel connectivity. Specifically, we treat all candidate EEG channels as a graph and define channel selection as the problem of node classification on a graph. Then we design a graph convolutional neural network (GCN) model for channels classification. Channels are selected based on the outputs of our GCN model. We evaluate our proposed GCN-based channel selection (GCN-CS) method on three MI datasets. On three datasets, GCN-CS achieves performance improvements by reducing the number of channels. Specifically, we achieve classification accuracies of 79.76% on Dataset 1, 89.14% on Dataset 2 and 87.96% on Dataset 3, which outperform competing methods significantly.

Feature learning framework based on EEG graph self-attention networks for motor imagery BCI systems.

Learning distinguishable features from raw EEG signals is crucial for accurate classification of motor imagery (MI) tasks. To incorporate spatial relationships between EEG sources, we developed a feature set based on an EEG graph. In this graph, EEG channels represent the nodes, with power spectral density (PSD) features defining their properties, and the edges preserving the spatial information. We designed an EEG based graph self-attention network (EGSAN) to learn low-dimensional embedding vector for EEG graph, which can be used as distinguishable features for motor imagery task classification. We evaluated our EGSAN model on two publicly available MI EEG datasets, each containing different types of motor imagery tasks. Our experiments demonstrate that our proposed model effectively extracts distinguishable features from EEG graphs, achieving significantly higher classification accuracies than existing state-of-the-art methods.

Classification of Motor Imagery Based on Multi-Scale Feature Extraction and the Channel-Temporal Attention Module.

Motor imagery (MI) is a popular paradigm for controlling electroencephalogram (EEG) based Brain-Computer Interface (BCI) systems. Many methods have been developed to attempt to accurately classify MI-related EEG activity. Recently, the development of deep learning has begun to draw increasing attention in the BCI research community because it does not need to use sophisticated signal preprocessing and can automatically extract features. In this paper, we propose a deep learning model for use in MI-based BCI systems. Our model makes use of a convolutional neural network based on a multi-scale and channel-temporal attention module (CTAM), which called MSCTANN. The multi-scale module is able to extract a large number of features, while the attention module includes both a channel attention module and a temporal attention module, which together allow the model to focus attention on the most important features extracted from the data. The multi-scale module and the attention module are connected by a residual module, which avoids the degradation of the network. Our network model is built from these three core modules, which combine to improve the recognition ability of the network for EEG signals. Our experimental results on three datasets (BCI competition IV 2a, III IIIa and IV 1) show that our proposed method has better performance than other state-of-the-art methods, with accuracy rates of 80.6%, 83.56% and 79.84%. Our model has stable performance in decoding EEG signals and achieves efficient classification performance while using fewer network parameters than other comparable state-of-the-art methods.

Neural decoding of music from the EEG.

Neural decoding models can be used to decode neural representations of visual, acoustic, or semantic information. Recent studies have demonstrated neural decoders that are able to decode accoustic information from a variety of neural signal types including electrocortiography (ECoG) and the electroencephalogram (EEG). In this study we explore how functional magnetic resonance imaging (fMRI) can be combined with EEG to develop an accoustic decoder. Specifically, we first used a joint EEG-fMRI paradigm to record brain activity while participants listened to music. We then used fMRI-informed EEG source localisation and a bi-directional long-term short term deep learning network to first extract neural information from the EEG related to music listening and then to decode and reconstruct the individual pieces of music an individual was listening to. We further validated our decoding model by evaluating its performance on a separate dataset of EEG-only recordings. We were able to reconstruct music, via our fMRI-informed EEG source analysis approach, with a mean rank accuracy of 71.8% ([Formula: see text], [Formula: see text]). Using only EEG data, without participant specific fMRI-informed source analysis, we were able to identify the music a participant was listening to with a mean rank accuracy of 59.2% ([Formula: see text], [Formula: see text]). This demonstrates that our decoding model may use fMRI-informed source analysis to aid EEG based decoding and reconstruction of acoustic information from brain activity and makes a step towards building EEG-based neural decoders for other complex information domains such as other acoustic, visual, or semantic information.

Sonic enhancement of virtual exhibits.

Museums have widely embraced virtual exhibits. However, relatively little attention is paid to how sound may create a more engaging experience for audiences. To begin addressing this lacuna, we conducted an online experiment to explore how sound influences the interest level, emotional response, and engagement of individuals who view objects within a virtual exhibit. As part of this experiment, we designed a set of different soundscapes, which we presented to participants who viewed museum objects virtually. We then asked participants to report their felt affect and level of engagement with the exhibits. Our results show that soundscapes customized to exhibited objects significantly enhance audience engagement. We also found that more engaged audience members were more likely to want to learn additional information about the object(s) they viewed and to continue viewing these objects for longer periods of time. Taken together, our findings suggest that virtual museum exhibits can improve visitor engagement through forms of customized soundscape design.

Bispectrum-based hybrid neural network for motor imagery classification.

BACKGROUND: The performance of motor imagery electroencephalogram (MI-EEG) decoding systems is easily affected by noise. As a higher-order spectra (HOS), the bispectrum is capable of suppressing Gaussian noise and increasing the signal-to-noise ratio of signals. However, the sum of logarithmic amplitudes (SLA) and the first order spectral moment (FOSM) features extracted from the bispectrum only use the numerical values of the bispectrum, ignoring the related information between different frequency bins. NEW METHOD: In this study, we proposed a novel framework, termed a bispectrum-based hybrid neural network (BHNN), to make full use of bispectrum for improving the performance of the MI-based brain-computer interface (BCI). Specifically, the BHNN consisted of a convolutional neural network (CNN), gated recurrent units (GRU), and squeeze-and-excitation (SE) modules. The SE modules and CNNs are first used to learn the deep relation between frequency bins of the bispectrum estimated from different time window segmentations of the MI-EEG. Then, we used GRU to seek the overlooked sequential information of the bispectrum. RESULTS: To validate the effectiveness of the proposed BHNN, three public BCI competition datasets were used in this study. The results demonstrated that the BHNN can achieve promising performance in decoding MI-EEG. COMPARISON WITH EXISTING METHODS: The statistical test results demonstrated that the proposed BHNN can significantly outperform other competing methods (p < =0.05). CONCLUSION: The proposed BHNN is a novel bispectrum-based neural network, which can enhance the decoding performance of MI-based BCIs.

Neural decoding of semantic concepts: a systematic literature review.

Objective.Semantic concepts are coherent entities within our minds. They underpin our thought processes and are a part of the basis for our understanding of the world. Modern neuroscience research is increasingly exploring how individual semantic concepts are encoded within our brains and a number of studies are beginning to reveal key patterns of neural activity that underpin specific concepts. Building upon this basic understanding of the process of semantic neural encoding, neural engineers are beginning to explore tools and methods for semantic decoding: identifying which semantic concepts an individual is focused on at a given moment in time from recordings of their neural activity. In this paper we review the current literature on semantic neural decoding.Approach.We conducted this review according to the Preferred Reporting Items for Systematic reviews and Meta-Analysis (PRISMA) guidelines. Specifically, we assess the eligibility of published peer-reviewed reports via a search of PubMed and Google Scholar. We identify a total of 74 studies in which semantic neural decoding is used to attempt to identify individual semantic concepts from neural activity.Main results.Our review reveals how modern neuroscientific tools have been developed to allow decoding of individual concepts from a range of neuroimaging modalities. We discuss specific neuroimaging methods, experimental designs, and machine learning pipelines that are employed to aid the decoding of semantic concepts. We quantify the efficacy of semantic decoders by measuring information transfer rates. We also discuss current challenges presented by this research area and present some possible solutions. Finally, we discuss some possible emerging and speculative future directions for this research area.Significance.Semantic decoding is a rapidly growing area of research. However, despite its increasingly widespread popularity and use in neuroscientific research this is the first literature review focusing on this topic across neuroimaging modalities and with a focus on quantifying the efficacy of semantic decoders.

SincNet-Based Hybrid Neural Network for Motor Imagery EEG Decoding.

It is difficult to identify optimal cut-off frequencies for filters used with the common spatial pattern (CSP) method in motor imagery (MI)-based brain-computer interfaces (BCIs). Most current studies choose filter cut-frequencies based on experience or intuition, resulting in sub-optimal use of MI-related spectral information in the electroencephalography (EEG). To improve information utilization, we propose a SincNet-based hybrid neural network (SHNN) for MI-based BCIs. First, raw EEG is segmented into different time windows and mapped into the CSP feature space. Then, SincNets are used as filter bank band-pass filters to automatically filter the data. Next, we used squeeze-and-excitation modules to learn a sparse representation of the filtered data. The resulting sparse data were fed into convolutional neural networks to learn deep feature representations. Finally, these deep features were fed into a gated recurrent unit module to seek sequential relations, and a fully connected layer was used for classification. We used the BCI competition IV datasets 2a and 2b to verify the effectiveness of our SHNN method. The mean classification accuracies (kappa values) of our SHNN method are 0.7426 (0.6648) on dataset 2a and 0.8349 (0.6697) on dataset 2b, respectively. The statistical test results demonstrate that our SHNN can significantly outperform other state-of-the-art methods on these datasets.

Feature Extraction Method Based on Filter Banks and Riemannian Tangent Space in Motor-Imagery BCI.

Optimal feature extraction for multi-category motor imagery brain-computer interfaces (MI-BCIs) is a research hotspot. The common spatial pattern (CSP) algorithm is one of the most widely used methods in MI-BCIs. However, its performance is adversely affected by variance in the operational frequency band and noise interference. Furthermore, the performance of CSP is not satisfactory when addressing multi-category classification problems. In this work, we propose a fusion method combining Filter Banks and Riemannian Tangent Space (FBRTS) in multiple time windows. FBRTS uses multiple filter banks to overcome the problem of variance in the operational frequency band. It also applies the Riemannian method to the covariance matrix extracted by the spatial filter to obtain more robust features in order to overcome the problem of noise interference. In addition, we use a One-Versus-Rest support vector machine (OVR-SVM) model to classify multi-category features. We evaluate our FBRTS method using BCI competition IV dataset 2a and 2b. The experimental results show that the average classification accuracy of our FBRTS method is 77.7% and 86.9% in datasets 2a and 2b respectively. By analyzing the influence of the different numbers of filter banks and time windows on the performance of our FBRTS method, we can identify the optimal number of filter banks and time windows. Additionally, our FBRTS method can obtain more distinctive features than the filter banks common spatial pattern (FBCSP) method in two-dimensional embedding space. These results show that our proposed method can improve the performance of MI-BCIs.

A Novel Classification Framework Using the Graph Representations of Electroencephalogram for Motor Imagery Based Brain-Computer Interface.

The motor imagery (MI) based brain-computer interfaces (BCIs) have been proposed as a potential physical rehabilitation technology. However, the low classification accuracy achievable with MI tasks is still a challenge when building effective BCI systems. We propose a novel MI classification model based on measurement of functional connectivity between brain regions and graph theory. Specifically, motifs describing local network structures in the brain are extracted from functional connectivity graphs. A graph embedding model called Ego-CNNs is then used to build a classifier, which can convert the graph from a structural representation to a fixed-dimensional vector for detecting critical structure in the graph. We validate our proposed method on four datasets, and the results show that our proposed method produces high classification accuracies in two-class classification tasks (92.8% for dataset 1, 93.4% for dataset 2, 96.5% for dataset 3, and 80.2% for dataset 4) and multiclass classification tasks (90.33% for dataset 1). Our proposed method achieves a mean Kappa value of 0.88 across nine participants, which is superior to other methods we compared it to. These results indicate that there is a local structural difference in functional connectivity graphs extracted under different motor imagery tasks. Our proposed method has great potential for motor imagery classification in future studies.

Feature selection method based on Menger curvature and LDA theory for a P300 brain-computer interface.

Brain-computer interface (BCI) systems decode electroencephalogram signals to establish a channel for direct interaction between the human brain and the external world without the need for muscle or nerve control. The P300 speller, one of the most widely used BCI applications, presents a selection of characters to the user and performs character recognition by identifying P300 event-related potentials from the EEG. Such P300-based BCI systems can reach good levels of accuracy but are difficult to use in day-to-day life due to redundancy and noisy signal. A room for improvement should be considered. We propose a novel hybrid feature selection method for the P300-based BCI system to address the problem of feature redundancy, which combines the Menger curvature and linear discriminant analysis. First, selected strategies are applied separately to a given dataset to estimate the gain for application to each feature. Then, each generated value set is ranked in descending order and judged by a predefined criterion to be suitable in classification models. The intersection of the two approaches is then evaluated to identify an optimal feature subset. The proposed method is evaluated using three public datasets, i.e., BCI Competition III dataset II, BNCI Horizon dataset, and EPFL dataset. Experimental results indicate that compared with other typical feature selection and classification methods, our proposed method has better or comparable performance. Additionally, our proposed method can achieve the best classification accuracy after all epochs in three datasets. In summary, our proposed method provides a new way to enhance the performance of the P300-based BCI speller.

Removal of physiological artifacts from simultaneous EEG and fMRI recordings.

OBJECTIVE: Simultaneous recording of the electroencephalogram (EEG) and functional magnetic resonance imaging (fMRI) allows a combination of eletrophysiological and haemodynamic information to be used to form a more complete picture of cerebral dynamics. However, EEG recorded within the MRI scanner is contaminated by both imaging artifacts and physiological artifacts. The majority of the techniques used to pre-process such EEG focus on removal of the imaging and balistocardiogram artifacts, with some success, but don't remove all other physiological artifacts. METHODS: We propose a new offline EEG artifact removal method based upon a combination of independent component analysis and fMRI-based head movement estimation to aid the removal of physiological artifacts from EEG recorded during EEG-fMRI recordings. Our method makes novel use of head movement trajectories estimated from the fMRI recording in order to assist with identifying physiological artifacts in the EEG and is designed to be used after removal of the fMRI imaging artifact from the EEG. RESULTS: We evaluate our method on EEG recorded during a joint EEG-fMRI session from healthy adult participants. Our method significantly reduces the influence of all types of physiological artifacts on the EEG. We also compare our method with a state-of-the-art physiological artifact removal method and demonstrate superior performance removing physiological artifacts. CONCLUSIONS: Our proposed method is able to remove significantly more physiological artifact components from the EEG, recorded during a joint EEG-fMRI session, than other state-of-the-art methods. SIGNIFICANCE: Our proposed method represents a marked improvement over current processing pipelines for removing physiological noise from EEG recorded during a joint EEG-fMRI session.

Enhancing P300 based character recognition performance using a combination of ensemble classifiers and a fuzzy fusion method.

BACKGROUND: P300-based brain-computer interfaces provide communication pathways without the need for muscle activity by recognizing electrical signals from the brain. The P300 speller is one of the most commonly used BCI applications, as it is very simple and reliable, and it is capable of reaching satisfactory communication performance. However, as with other BCIs, it remains a challenge to improve the P300 speller's performance to increase its practical usability. NEW METHODS: In this study, we propose a novel multi-feature subset fuzzy fusion (MSFF) framework for the P300 speller to recognize the users' spelling intention. This method includes two parts: 1) feature selection by the Lasso algorithm and feature division; 2) the construction of ensemble LDA classifiers and the fuzzy fusion of those classifiers to recognize user intention. RESULTS: The proposed framework is evaluated in three public datasets and achieves an average accuracy of 100% after 4 epochs for BCI Competition II Dataset IIb, 96% for BCI Competition III dataset II and 98.3% for the BNCI Horizon Dataset. It indicates that the proposed MSFF method can make use of temporal information of signals and helps to enhance classification performance. COMPARISON WITH EXISTING METHODS: The proposed MSFF method yields better or comparable performance than previously reported machine learning algorithms. CONCLUSIONS: The proposed MSFF method is able to improve the performance of P300-based BCIs.

Optimization of Model Training Based on Iterative Minimum Covariance Determinant In Motor-Imagery BCI.

The common spatial patterns (CSP) algorithm is one of the most frequently used and effective spatial filtering methods for extracting relevant features for use in motor imagery brain-computer interfaces (MI-BCIs). However, the inherent defect of the traditional CSP algorithm is that it is highly sensitive to potential outliers, which adversely affects its performance in practical applications. In this work, we propose a novel feature optimization and outlier detection method for the CSP algorithm. Specifically, we use the minimum covariance determinant (MCD) to detect and remove outliers in the dataset, then we use the Fisher score to evaluate and select features. In addition, in order to prevent the emergence of new outliers, we propose an iterative minimum covariance determinant (IMCD) algorithm. We evaluate our proposed algorithm in terms of iteration times, classification accuracy and feature distribution using two BCI competition datasets. The experimental results show that the average classification performance of our proposed method is 12% and 22.9% higher than that of the traditional CSP method in two datasets ([Formula: see text]), and our proposed method obtains better performance in comparison with other competing methods. The results show that our method improves the performance of MI-BCI systems.

How to build a fast and accurate code-modulated brain-computer interface.

Objective.In the last decade, the advent of code-modulated brain-computer interfaces (BCIs) has allowed the implementation of systems with high information transfer rates (ITRs) and increased the possible practicality of such interfaces. In this paper, we evaluate the effect of different numbers of targets in the stimulus display, modulation sequences generators, and signal processing algorithms on the accuracy and ITR of code-modulated BCIs.Approach.We use both real and simulated electroencephalographic (EEG) data, to evaluate these parameters and methods. Then, we compared numerous different setups to assess their performance and identify the best configurations. We also evaluated the dependability of our simulated evaluation approach.Main results.Our results show that Golay, almost perfect, and deBruijn sequence-based visual stimulus modulations provide the best results, significantly outperforming the commonly used m-sequences in all cases. We conclude that artificial neural network processing algorithms offer the best processing pipeline for this type of BCI, achieving a maximum classification accuracy of 94.7% on real EEG data while obtaining a maximum ITR of 127.2 bits min-1in a simulated 64-target system.Significance.We used a simulated framework that demonstrated previously unattainable flexibility and convenience while staying reasonably realistic. Furthermore, our findings suggest several new considerations which can be used to guide further code-based BCI development.